{
  "nbformat": 4,
  "nbformat_minor": 2,
  "metadata": {},
  "cells": [
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "   <a target=\"_blank\" href=\"https://labelbox.com\" ><img src=\"https://labelbox.com/blog/content/images/2021/02/logo-v4.svg\" width=256/></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "<td>\n",
        "<a href=\"https://colab.research.google.com/github/Labelbox/labelbox-python/blob/master/examples/prediction_upload/conversational_predictions.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"></a>\n",
        "</td>\n",
        "\n",
        "<td>\n",
        "<a href=\"https://github.com/Labelbox/labelbox-python/tree/master/examples/prediction_upload/conversational_predictions.ipynb\" target=\"_blank\"><img\n",
        "src=\"https://img.shields.io/badge/GitHub-100000?logo=github&logoColor=white\" alt=\"GitHub\"></a>\n",
        "</td>"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Conversational Text Prediction Import\n",
        "* This notebook will provide examples of each supported prediction type for conversational text assets, and also cover MAL and Label Import methods:\n",
        "\n",
        "Suported annotations that can be uploaded through the SDK\n",
        "\n",
        "* Classification Radio \n",
        "* Classification Checklist \n",
        "* Classification Free Text \n",
        "* NER\n",
        "\n",
        "**Not** supported annotations\n",
        "\n",
        "* Bouding box \n",
        "* Polygon \n",
        "* Point\n",
        "* Polyline \n",
        "* Segmentation Mask \n",
        "* Relationships\n",
        "\n",
        "\n"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "## Setup"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "!pip install -q \"labelbox[data]\""
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "import labelbox as lb\n",
        "import uuid\n",
        "import labelbox.types as lb_types"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Replace with your API key"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Add your api key\n",
        "API_KEY = \"\"\n",
        "client = lb.Client(api_key=API_KEY)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Supported Predictions "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "########### Radio Classification ###########\n",
        "\n",
        "# Python annotation\n",
        "radio_prediction = lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_question\", \n",
        "    value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = \"second_radio_answer\", confidence=0.5))\n",
        ")\n",
        "\n",
        "# NDJSON\n",
        "radio_prediction_ndjson = {\n",
        "  \"name\": \"radio_question\",\n",
        "  \"answer\": {\"name\": \"second_radio_answer\", \"confidence\": 0.5}\n",
        "} "
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "\n",
        "# message based classifications\n",
        "ner_prediction = lb_types.ObjectAnnotation(\n",
        "    name=\"ner\",\n",
        "    confidence=0.5,\n",
        "    value=lb_types.ConversationEntity(\n",
        "        start=0,\n",
        "        end=8,\n",
        "        message_id=\"4\"\n",
        "    )\n",
        ")\n",
        "\n",
        "ner_prediction_ndjson = { \n",
        "        \"name\": \"ner\",\n",
        "        \"confidence\":0.5,\n",
        "        \"location\": { \n",
        "            \"start\": 0, \n",
        "            \"end\": 8 \n",
        "        },\n",
        "        \"messageId\": \"4\"\n",
        "    }"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "##### Classification free text #####\n",
        "# Confidence scores are not supported for text predictions\n",
        "\n",
        "text_prediction = lb_types.ClassificationAnnotation(\n",
        "    name=\"text_convo\",\n",
        "    value=lb_types.Text(answer=\"the answer to the text questions are right here\"),\n",
        "    message_id=\"0\"\n",
        ")\n",
        "\n",
        "\n",
        "text_prediction_ndjson = {\n",
        "    \"name\": \"text_convo\",\n",
        "    \"answer\": \"the answer to the text questions are right here\",\n",
        "    \"messageId\": \"0\"\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "\n",
        "##### Checklist Classification ####### \n",
        "\n",
        "checklist_prediction= lb_types.ClassificationAnnotation(\n",
        "  name=\"checklist_convo\", # must match your ontology feature\"s name\n",
        "  value=lb_types.Checklist(\n",
        "      answer = [\n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"first_checklist_answer\", confidence=0.5\n",
        "        ), \n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"second_checklist_answer\", confidence=0.5\n",
        "        )\n",
        "      ]\n",
        "    ),\n",
        "  message_id=\"2\"\n",
        " )\n",
        "\n",
        "\n",
        "checklist_prediction_ndjson = {\n",
        "    \"name\": \"checklist_convo\",\n",
        "    \"answers\": [\n",
        "        {\"name\": \"first_checklist_answer\", \"confidence\":0.5},\n",
        "        {\"name\": \"second_checklist_answer\", \"confidence\":0.5}\n",
        "    ],\n",
        "    \"messageId\": \"2\"\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "######## Radio Classification ######\n",
        "\n",
        "radio_prediction = lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_convo\", \n",
        "    value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = \"first_radio_answer\", confidence=0.5)),\n",
        "    message_id=\"0\"\n",
        ")\n",
        "\n",
        "radio_prediction_ndjson = {\n",
        "    \"name\": \"radio_convo\",\n",
        "    \"answer\": {\n",
        "        \"name\": \"first_radio_answer\", \"confidence\":0.5\n",
        "    },\n",
        "    \"messageId\": \"0\",\n",
        "}"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "# ############ global nested classifications ###########\n",
        "\n",
        "# Message based\n",
        "nested_checklist_prediction = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_checklist_question\",\n",
        "  message_id=\"10\",\n",
        "  value=lb_types.Checklist(\n",
        "    answer=[lb_types.ClassificationAnswer(\n",
        "      name=\"first_checklist_answer\",\n",
        "      confidence=0.5, # Confidence scores should be added to the answer \n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_checklist_question\",\n",
        "          value=lb_types.Checklist(\n",
        "            answer=[lb_types.ClassificationAnswer(\n",
        "            name=\"first_sub_checklist_answer\",\n",
        "            confidence=0.5 # Confidence scores should be added to the answer \n",
        "          )]\n",
        "        ))\n",
        "      ]\n",
        "    )]\n",
        "  )\n",
        ")\n",
        "# Message based\n",
        "nested_checklist_prediction_ndjson = {\n",
        "  \"name\": \"nested_checklist_question\",\n",
        "  \"messageId\": \"10\",\n",
        "  \"answer\": [{\n",
        "      \"name\": \"first_checklist_answer\", \n",
        "      \"confidence\": 0.5, # Confidence scores should be added to the answer \n",
        "      \"classifications\" : [\n",
        "        {\n",
        "          \"name\": \"sub_checklist_question\", \n",
        "          \"answer\": {\n",
        "            \"name\": \"first_sub_checklist_answer\", \n",
        "            \"confidence\": 0.5, # Confidence scores should be added to the answer \n",
        "          }\n",
        "        }          \n",
        "      ]         \n",
        "  }]\n",
        "}\n",
        "# Global\n",
        "nested_radio_prediction = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_radio_question\",\n",
        "  value=lb_types.Radio(\n",
        "    answer=lb_types.ClassificationAnswer(\n",
        "      name=\"first_radio_answer\",\n",
        "      confidence=0.5, # Confidence scores should be added to the answer \n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_radio_question\",\n",
        "          value=lb_types.Radio(\n",
        "            answer=lb_types.ClassificationAnswer(\n",
        "              name=\"first_sub_radio_answer\",\n",
        "              confidence=0.5 # Confidence scores should be added to the answer\n",
        "            )\n",
        "          )\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  )\n",
        ")\n",
        "#Global\n",
        "nested_radio_prediction_ndjson = {\n",
        "  \"name\": \"nested_radio_question\",\n",
        "  \"answer\": {\n",
        "      \"name\": \"first_radio_answer\",\n",
        "      \"confidence\": 0.5,\n",
        "      \"classifications\": [{\n",
        "          \"name\":\"sub_radio_question\",\n",
        "          \"answer\": { \"name\" : \"first_sub_radio_answer\", \n",
        "                     \"confidence\": 0.5}\n",
        "        }]\n",
        "    }\n",
        "}\n",
        "\n"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 1: Import data rows into Catalog "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Create one Labelbox dataset\n",
        "\n",
        "global_key = \"conversation-1.json\"\n",
        "\n",
        "asset = {\n",
        "    \"row_data\": \"https://storage.googleapis.com/labelbox-developer-testing-assets/conversational_text/1000-conversations/conversation-1.json\",\n",
        "    \"global_key\": global_key\n",
        "}\n",
        "\n",
        "dataset = client.create_dataset(name=\"conversational_annotation_import_demo_dataset\")\n",
        "task = dataset.create_data_rows([asset])\n",
        "task.wait_till_done()\n",
        "print(\"Errors:\", task.errors)\n",
        "print(\"Failed data rows: \", task.failed_data_rows)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 2: Create/select an Ontology for your model predictions\n",
        "Your project should have the correct ontology setup with all the tools and classifications supported for your annotations, and the tool names and classification instructions should match the name/instructions fields in your annotations to ensure the correct feature schemas are matched."
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "ontology_builder = lb.OntologyBuilder(\n",
        "  tools=[ \n",
        "    lb.Tool(tool=lb.Tool.Type.NER,name=\"ner\")\n",
        "    ], \n",
        "  classifications=[ \n",
        "    lb.Classification( \n",
        "      class_type=lb.Classification.Type.TEXT,\n",
        "      scope=lb.Classification.Scope.INDEX,          \n",
        "      name=\"text_convo\"), \n",
        "    lb.Classification( \n",
        "      class_type=lb.Classification.Type.CHECKLIST, \n",
        "      scope=lb.Classification.Scope.INDEX,                     \n",
        "      name=\"checklist_convo\", \n",
        "      options=[\n",
        "        lb.Option(value=\"first_checklist_answer\"),\n",
        "        lb.Option(value=\"second_checklist_answer\")            \n",
        "      ]\n",
        "    ), \n",
        "    lb.Classification( \n",
        "      class_type=lb.Classification.Type.RADIO, \n",
        "      name=\"radio_convo\", \n",
        "      scope=lb.Classification.Scope.INDEX,          \n",
        "      options=[\n",
        "        lb.Option(value=\"first_radio_answer\"),\n",
        "        lb.Option(value=\"second_radio_answer\")\n",
        "      ]\n",
        "    ),\n",
        "    lb.Classification(\n",
        "      class_type=lb.Classification.Type.CHECKLIST,\n",
        "      name=\"nested_checklist_question\",\n",
        "      scope = lb.Classification.Scope.INDEX,\n",
        "      options=[\n",
        "          lb.Option(\"first_checklist_answer\",\n",
        "            options=[\n",
        "              lb.Classification(\n",
        "                  class_type=lb.Classification.Type.CHECKLIST,\n",
        "                  name=\"sub_checklist_question\", \n",
        "                  options=[lb.Option(\"first_sub_checklist_answer\")]\n",
        "              )\n",
        "          ])\n",
        "      ]\n",
        "    ),\n",
        "    lb.Classification(\n",
        "        class_type=lb.Classification.Type.RADIO,\n",
        "        name=\"nested_radio_question\",\n",
        "        scope = lb.Classification.Scope.GLOBAL,\n",
        "        options=[\n",
        "            lb.Option(\"first_radio_answer\",\n",
        "                options=[\n",
        "                    lb.Classification(\n",
        "                        class_type=lb.Classification.Type.RADIO,\n",
        "                        name=\"sub_radio_question\",\n",
        "                        options=[lb.Option(\"first_sub_radio_answer\")]\n",
        "                    )\n",
        "                ])\n",
        "          ]\n",
        "    )\n",
        "  ]\n",
        ")\n",
        "\n",
        "ontology = client.create_ontology(\"Ontology Conversation Annotations\", ontology_builder.asdict())"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 3: Create a Mode and Model Run "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# create Model\n",
        "model = client.create_model(name=\"Conversational_model_run_\"+ str(uuid.uuid4()), \n",
        "                            ontology_id=ontology.uid)\n",
        "# create Model Run\n",
        "model_run = model.create_model_run(\"iteration 1\")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 4: Send data rows to the Model Run "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "model_run.upsert_data_rows(global_keys=[global_key])"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 5: Create the predictions payload\n",
        "Create the prediction payload using the snippets of code in the **Supported Predcitions** section\n",
        "\n",
        "Labelbox support two formats for the annotations payload: NDJSON and Python Annotation types. Both are described below to compose your annotations into Labels attached to the data rows.\n",
        "\n",
        "The resulting payload should have exactly the same content for annotations that are supported by both"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "Python annotations"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "label_prediction = []\n",
        "label_prediction.append(\n",
        "  lb_types.Label(\n",
        "    data=lb_types.ConversationData(\n",
        "      global_key=global_key\n",
        "    ),\n",
        "    annotations=[\n",
        "      ner_prediction,\n",
        "      checklist_prediction,\n",
        "      text_prediction,\n",
        "      radio_prediction,\n",
        "      nested_checklist_prediction,\n",
        "      nested_radio_prediction\n",
        "    ]\n",
        "  )\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "if using NDJSON : "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "label_prediction_ndjson = []\n",
        "for annotations in [\n",
        "    ner_prediction_ndjson,\n",
        "    text_prediction_ndjson,\n",
        "    checklist_prediction_ndjson,\n",
        "    radio_prediction_ndjson,\n",
        "    nested_checklist_prediction_ndjson,\n",
        "    nested_radio_prediction_ndjson\n",
        "    ]:\n",
        "  annotations.update({\n",
        "      \"dataRow\": {\n",
        "          \"globalKey\": global_key\n",
        "      }\n",
        "  })\n",
        "  label_prediction_ndjson.append(annotations)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 6: Upload the predictions payload to the Model Run"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# Upload the prediction label to the Model Run\n",
        "upload_job_prediction = model_run.add_predictions(\n",
        "    name=\"prediction_upload_job\"+str(uuid.uuid4()),\n",
        "    predictions=label_prediction)\n",
        "\n",
        "# Errors will appear for annotation uploads that failed.\n",
        "print(\"Errors:\", upload_job_prediction.errors)\n",
        "print(\"Status of uploads: \", upload_job_prediction.statuses)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "## Step 7 : Send annotations to the Model Run "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "7.1 Create a labelbox project"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "project = client.create_project(name=\"Conversational Text Prediction Import Demo\",                                    \n",
        "                                    media_type=lb.MediaType.Conversational)\n",
        "project.setup_editor(ontology)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "7.2 Create a batch to send to the project"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "project.create_batch(\n",
        "  \"batch_convo_prediction_demo\", # Each batch in a project must have a unique name\n",
        "  global_keys=[global_key], # Paginated collection of data row objects, list of data row ids or global keys\n",
        "  priority=5 # priority between 1(Highest) - 5(lowest)\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "7.3 Create the annotations payload"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "ner_annotation = lb_types.ObjectAnnotation(\n",
        "    name=\"ner\",\n",
        "    value=lb_types.ConversationEntity(\n",
        "        start=0,\n",
        "        end=8,\n",
        "        message_id=\"4\"\n",
        "    )\n",
        ")\n",
        "\n",
        "text_annotation = lb_types.ClassificationAnnotation(\n",
        "    name=\"text_convo\",\n",
        "    value=lb_types.Text(answer=\"the answer to the text questions are right here\"),\n",
        "    message_id=\"0\"\n",
        ")\n",
        "\n",
        "checklist_annotation= lb_types.ClassificationAnnotation(\n",
        "  name=\"checklist_convo\", # must match your ontology feature\"s name\n",
        "  value=lb_types.Checklist(\n",
        "      answer = [\n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"first_checklist_answer\"\n",
        "        ), \n",
        "        lb_types.ClassificationAnswer(\n",
        "            name = \"second_checklist_answer\"\n",
        "        )\n",
        "      ]\n",
        "    ),\n",
        "  message_id=\"2\"\n",
        " )\n",
        "\n",
        "radio_annotation = lb_types.ClassificationAnnotation(\n",
        "    name=\"radio_convo\", \n",
        "    value=lb_types.Radio(answer = lb_types.ClassificationAnswer(name = \"first_radio_answer\")),\n",
        "    message_id=\"0\"\n",
        ")\n",
        "\n",
        "\n",
        "nested_checklist_annotation = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_checklist_question\",\n",
        "  message_id=\"10\",\n",
        "  value=lb_types.Checklist(\n",
        "    answer=[lb_types.ClassificationAnswer(\n",
        "      name=\"first_checklist_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_checklist_question\",\n",
        "          value=lb_types.Checklist(\n",
        "            answer=[lb_types.ClassificationAnswer(\n",
        "            name=\"first_sub_checklist_answer\"\n",
        "          )]\n",
        "        ))\n",
        "      ]\n",
        "    )]\n",
        "  )\n",
        ")\n",
        "\n",
        "nested_radio_annotation = lb_types.ClassificationAnnotation(\n",
        "  name=\"nested_radio_question\",\n",
        "  value=lb_types.Radio(\n",
        "    answer=lb_types.ClassificationAnswer(\n",
        "      name=\"first_radio_answer\",\n",
        "      classifications=[\n",
        "        lb_types.ClassificationAnnotation(\n",
        "          name=\"sub_radio_question\",\n",
        "          value=lb_types.Radio(\n",
        "            answer=lb_types.ClassificationAnswer(\n",
        "              name=\"first_sub_radio_answer\"\n",
        "            )\n",
        "          )\n",
        "        )\n",
        "      ]\n",
        "    )\n",
        "  )\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "7.4 Create the label object"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "label = []\n",
        "label.append(\n",
        "  lb_types.Label(\n",
        "    data=lb_types.ConversationData(\n",
        "      global_key=global_key\n",
        "    ),\n",
        "    annotations=[\n",
        "      ner_annotation,\n",
        "      text_annotation,\n",
        "      checklist_annotation,\n",
        "      radio_annotation,\n",
        "      nested_radio_annotation,\n",
        "      nested_checklist_annotation\n",
        "    ]\n",
        "  )\n",
        ")"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "7.5 Upload annotations to the project using Label Import"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "upload_job_annotation = lb.LabelImport.create_from_objects(\n",
        "    client = client,\n",
        "    project_id = project.uid,\n",
        "    name=\"text_label_import_job\"+ str(uuid.uuid4()),\n",
        "    labels=label)\n",
        "\n",
        "upload_job_annotation.wait_until_done()\n",
        "# Errors will appear for annotation uploads that failed.\n",
        "print(\"Errors:\", upload_job_annotation.errors)\n",
        "print(\"Status of uploads: \", upload_job_annotation.statuses)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "7.6 Send the annotations to the Model Run "
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# get the labels id from the project\n",
        "model_run.upsert_labels(project_id=project.uid)"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    },
    {
      "metadata": {},
      "source": [
        "# Option deletions for cleanup"
      ],
      "cell_type": "markdown"
    },
    {
      "metadata": {},
      "source": [
        "# project.delete()\n",
        "# dataset.delete()"
      ],
      "cell_type": "code",
      "outputs": [],
      "execution_count": null
    }
  ]
}